Skip to content

Fix BPE bonus materials #561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Mar 8, 2025
Merged

Fix BPE bonus materials #561

merged 8 commits into from
Mar 8, 2025

Conversation

rasbt
Copy link
Owner

@rasbt rasbt commented Mar 8, 2025

Fixes issues with the BPE tokenizer to correctly handle edge cases.

Fixes #558

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@rasbt rasbt marked this pull request as draft March 8, 2025 17:01
@rasbt rasbt marked this pull request as ready for review March 8, 2025 20:44
@rasbt rasbt mentioned this pull request Mar 8, 2025
Closed
@rasbt rasbt merged commit f63f04d into main Mar 8, 2025
13 checks passed
@rasbt rasbt deleted the fix-bpe branch March 8, 2025 23:21
@d-kleine
Copy link
Contributor

d-kleine commented Mar 9, 2025 •
edited
Loading

@rasbt About the lowest possible merge rank defined as min_rank = 1_000_000_000, I am not sure if inf would be a better choice (just like in the original BPE implementation) to keep the code resilient. The value 1_000_000_000 might look very specific to users, although it's arbitrary. Also, with multi-modality and growing vocabulary sizes, adjacent token combinations of 1_000_000_000 might be reached at some point in the future, possible breaking the code then.

LLMs-from-scratch/ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb

Lines 607 to 613 in f63f04d

" min_rank = 1_000_000_000\n",
" bigram = None\n",
" for p in pairs:\n",
" r = self.bpe_ranks.get(p, 1_000_000_000)\n",
" if r < min_rank:\n",
" min_rank = r\n",
" bigram = p\n",

@rasbt rasbt mentioned this pull request Mar 9, 2025
Merged
@rasbt
Copy link
Owner Author

rasbt commented Mar 9, 2025

yes, I agree!

rasbt added a commit that referenced this pull request Aug 17, 2025
@rasbt
Fix BPE bonus materials (#561)
6aec412 
* Fix BPE bonus materials

* fix bpe implementation

* update

* Add 'Hello, world. Is this-- a test?' test case

* update link to test file

* update path handling

* update path handling

* fix pytest paths
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Encoding issue with "Hello,"
2 participants
@rasbt @d-kleine